DROPS

Document

DOI: 10.4230/LIPIcs.SoCG.2016.65

Geometric Models for Musical Audio Data

Authors: Paul Bendich, Ellen Gasparovic, John Harer, and Christopher Tralie

Published in: LIPIcs, Volume 51, 32nd International Symposium on Computational Geometry (SoCG 2016)

Abstract

We study the geometry of sliding window embeddings of audio features that summarize perceptual information about audio, including its pitch and timbre. These embeddings can be viewed as point clouds in high dimensions, and we add structure to the point clouds using a cover tree with adaptive thresholds based on multi-scale local principal component analysis to automatically assign points to clusters. We connect neighboring clusters in a scaffolding graph, and we use knowledge of stratified space structure to refine our estimates of dimension in each cluster, demonstrating in our music applications that choruses and verses have higher dimensional structure, while transitions between them are lower dimensional. We showcase our technique with an interactive web-based application powered by Javascript and WebGL which plays music synchronized with a principal component analysis embedding of the point cloud down to 3D. We also render the clusters and the scaffolding on top of this projection to visualize the transitions between different sections of the music.

Cite as

Paul Bendich, Ellen Gasparovic, John Harer, and Christopher Tralie. Geometric Models for Musical Audio Data. In 32nd International Symposium on Computational Geometry (SoCG 2016). Leibniz International Proceedings in Informatics (LIPIcs), Volume 51, pp. 65:1-65:5, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2016)

Copy BibTex To Clipboard

@InProceedings{bendich_et_al:LIPIcs.SoCG.2016.65,
  author =	{Bendich, Paul and Gasparovic, Ellen and Harer, John and Tralie, Christopher},
  title =	{{Geometric Models for Musical Audio Data}},
  booktitle =	{32nd International Symposium on Computational Geometry (SoCG 2016)},
  pages =	{65:1--65:5},
  series =	{Leibniz International Proceedings in Informatics (LIPIcs)},
  ISBN =	{978-3-95977-009-5},
  ISSN =	{1868-8969},
  year =	{2016},
  volume =	{51},
  editor =	{Fekete, S\'{a}ndor and Lubiw, Anna},
  publisher =	{Schloss Dagstuhl -- Leibniz-Zentrum f{\"u}r Informatik},
  address =	{Dagstuhl, Germany},
  URL =		{https://drops-dev.dagstuhl.de/entities/document/10.4230/LIPIcs.SoCG.2016.65},
  URN =		{urn:nbn:de:0030-drops-59577},
  doi =		{10.4230/LIPIcs.SoCG.2016.65},
  annote =	{Keywords: Geometric Models, Audio Analysis, High Dimensional Data Analysis, Stratified Space Models}
}

Document

DOI: 10.4230/LIPIcs.SoCG.2016.71

High-Dimensional Geometry of Sliding Window Embeddings of Periodic Videos

Authors: Christopher Tralie

Published in: LIPIcs, Volume 51, 32nd International Symposium on Computational Geometry (SoCG 2016)

Abstract

We explore the high dimensional geometry of sliding windows of periodic videos. Under a reasonable model for periodic videos, we show that the sliding window is necessary to disambiguate all states within a period, and we show that a video embedding with a sliding window of an appropriate dimension lies on a topological loop along a hypertorus. This hypertorus has an independent ellipse for each harmonic of the motion. Natural motions with sharp transitions from foreground to background have many harmonics and are hence in higher dimensions, so linear subspace projections such as PCA do not accurately summarize the geometry of these videos. Noting this, we invoke tools from topological data analysis and cohomology to parameterize motions in high dimensions with circular coordinates after the embeddings. We show applications to videos in which there is obvious periodic motion and to videos in which the motion is hidden.

Cite as

Christopher Tralie. High-Dimensional Geometry of Sliding Window Embeddings of Periodic Videos. In 32nd International Symposium on Computational Geometry (SoCG 2016). Leibniz International Proceedings in Informatics (LIPIcs), Volume 51, pp. 71:1-71:5, Schloss Dagstuhl – Leibniz-Zentrum für Informatik (2016)

Copy BibTex To Clipboard

@InProceedings{tralie:LIPIcs.SoCG.2016.71,
  author =	{Tralie, Christopher},
  title =	{{High-Dimensional Geometry of Sliding Window Embeddings of Periodic Videos}},
  booktitle =	{32nd International Symposium on Computational Geometry (SoCG 2016)},
  pages =	{71:1--71:5},
  series =	{Leibniz International Proceedings in Informatics (LIPIcs)},
  ISBN =	{978-3-95977-009-5},
  ISSN =	{1868-8969},
  year =	{2016},
  volume =	{51},
  editor =	{Fekete, S\'{a}ndor and Lubiw, Anna},
  publisher =	{Schloss Dagstuhl -- Leibniz-Zentrum f{\"u}r Informatik},
  address =	{Dagstuhl, Germany},
  URL =		{https://drops-dev.dagstuhl.de/entities/document/10.4230/LIPIcs.SoCG.2016.71},
  URN =		{urn:nbn:de:0030-drops-59630},
  doi =		{10.4230/LIPIcs.SoCG.2016.71},
  annote =	{Keywords: Video Processing, High Dimensional Geometry, Circular Coordinates, Nonlinear Time Series}
}

Search Results

Documents authored by Tralie, Christopher

Geometric Models for Musical Audio Data

Abstract

Cite as

High-Dimensional Geometry of Sliding Window Embeddings of Periodic Videos

Abstract

Cite as

Thanks for your feedback!

Could not send message